Tilt-shift correction for camera arrays

ABSTRACT

An imaging system comprising of a set of cameras, the cameras having a configured relative camera arrangement; and each camera of the set of cameras comprising at least one corrective optical system. The system and method may comprise of multi-camera variations for coordinated alignment in multi-camera variations and/or split-field optical systems.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation Application of U.S. patentapplication Ser. No. 15/976,850, filed on 10 May 2018, which claims thebenefit of U.S. Provisional Application No. 62/504,501, filed on 10 May2017, both of which are incorporated in their entirety by thisreference.

TECHNICAL FIELD

This invention relates generally to the field of multi-camera systems,and more specifically to a new and useful system and method foroutfitting an environment with a camera array.

BACKGROUND

There is an emerging field of technology for sensor-driven monitoringwithin an environment to drive various services and applications. As oneexample, recent technology developments have seen initial proposedsystems directed towards enabling automatic checkout within a store byusing computer vision, sometimes in combination with other forms ofsensing. Many such implementations are limited in their wide spread usebecause of numerous factors including operational complexity andprohibitive system installation costs.

As one example, installing a multi-camera surveillance system in anenvironment such as a building can be a large project. The equipmentinvolved can be expensive and often includes complex and expensivecameras. Similarly, the installation process or even the feasibility ofinstallation is complicated when considering installation into anexisting store with existing infrastructure (e.g., shelves, lights,venting, etc.).

Additionally, a high level of training is required for workers toinstall a highly customized system. Even with training, setting up asystem can be a long process since installing, aiming, calibrating, andconfiguring each camera can be an involved process. Maintenance of thesystem is similarly expensive and complex. Furthermore, when in activeuse, the multi-camera surveillance systems can generate large amounts ofdata that can, ironically, limit their uses. Thus, there is a need inthe multi-camera system field to create a new and useful system andmethod for ubiquitous video monitoring across an environment. Thisinvention provides such a new and useful system and method.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of an instantiation of the system;

FIG. 2 is a network diagram representation of a series of cameramodules;

FIG. 3 is a schematic representation of an instantiation of the system;

FIGS. 4A and 4B are schematic representations of exemplaryconfigurations of a suspended camera module;

FIG. 5 is a schematic representation of an exemplary configuration oftwo rows of camera modules aligned within a shelving aisle;

FIG. 6 is a schematic representation of an exemplary configuration of asuspended camera module;

FIG. 7 is a schematic representation of an exemplary camera module railenclosure body;

FIG. 8 is a schematic representation of a suspended camera module;

FIG. 9A is a bottom schematic view of a camera module body compatiblewith ceiling tiles;

FIG. 9B is a top schematic view of a camera module body usable within aceiling tile structural system;

FIG. 10 is a schematic representation of camera and camera modulespacing;

FIG. 11 is a schematic representation of an exemplary single-pointcamera module;

FIG. 12 is a schematic representation of mounted camera modules;

FIG. 13 is a schematic representation of an alternative mounting systemfor camera modules;

FIGS. 14A and 14B are schematic representation of camera modules,redundant power and network connectivity, and modular use of camerasub-system units;

FIGS. 15A-15E are exemplary cross-sectional profiles of camera modules;

FIG. 16 is a schematic representation of a flexible camera module;

FIG. 17 is a diagram representation of a lens used in augmenting opticalobservation of a shelf;

FIG. 18A is a schematic representation of camera coverage and cameracoverage overlap for a shelf as collected by cameras;

FIGS. 18B-18D are schematic representations of camera coverage andcamera coverage overlap for a shelf as collected by cameras after use ofa corrective optical system;

FIG. 19A-9C are exemplary lighting integration form factors of differentcamera module variations;

FIGS. 20 and 21 are schematic representations of camera module componentarchitecture for operational redundancy;

FIG. 22 is a schematic representation of power and network connectivitycabling connecting camera modules;

FIG. 23 is a schematic representation of a monitoring network branch;

FIG. 24 is a schematic representation of a connection port;

FIG. 25 is a schematic representation of an exemplary wire managementsolution; and

FIGS. 26-29 are generalized schematic representation of variations ofsplit-field optical systems.

DESCRIPTION OF THE EMBODIMENTS

The following description of the embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.

1. Overview

A system and method for ubiquitous video monitoring across anenvironment functions to enable a network of cameras to be efficientlyinstalled and operated for collecting media, in particular, video andstill images. The collected media of the system is preferably used inlarge-scale computer vision (CV) driven applications. The system andmethod preferably involve the setup of multiple camera modules along, orsuspended from, the ceiling region of an environment. The system andmethod may alternatively include installation across other regionsand/or infrastructure of an environment. A particular attribute of thesystem and method is the use of easily installed camera modules, whichcan be daisy-chained or otherwise connected to form a monitoringnetwork. When used within a commercial space or suitable environment,the monitoring network can be distributed across the ceiling and otherregions for detailed visual monitoring. Other installation approachesmay additionally or alternatively be used with the system and method.

The system is primarily described as a network for multiple videomonitoring nodes. As used herein, ‘video’ might mean a common videoformat such as NTSC, PLA, MPEG, 4K, HEVC etc. Video may also mean asequence of individual still images whose place in time is monotonic.Those images may similarly be in a wide range of formats, including Raw,JPG, PDF, GIF, BMP, HEIF, WebP, and/or other suitable media formats.

The system could additionally be applied to a single camera and/or audiomonitoring and may additionally include other forms of sensing that maybe used independently or in combination with video monitoring. Thesemight include lidar, sonar, infrared, ultraviolet, radio frequency,capacitance, magnetic, scales, pressure, volume, climate sensing, andthe like.

Similarly, the system and method may additionally or alternatively beused in distributing other computer input or output devices across anenvironment. The system and method can be used in the collection ofsensor data and/or generation of an output in addition to or as analternative to video and/or image data. Other forms of devices such asmicrophones, bluetooth beacons, speakers, projectors, and other suitabledevices could additionally or alternatively be integrated into systemmodules that may be installed across an environment. Herein, the systemand method are primarily described as it relates to image-based videomonitoring.

The system and method are preferably used for applications that utilizeubiquitous monitoring across an environment. Herein, ubiquitousmonitoring (or more specifically ubiquitous video monitoring)characterizes pervasive sensor monitoring across regions of interest inan environment. Ubiquitous monitoring will generally have a largecoverage area that is preferably substantially continuous thoughdiscontinuities of a region may be supported. Additionally, monitoringmay monitor with a substantially uniform data resolution.

Large coverage, in one example, can be characterized as having greaterthan 95% of surface area of interest monitored. In a shoppingenvironment this can mean the shelves and product displays as well asthe shopping floor are monitored.

Substantial uniform data resolution preferably describes a sensingconfiguration where the variability of image resolution of differentareas in the environment is within a target range. In the exemplary caseof automatic checkout CV-driven applications, the target range for imageresolution is sufficient to resolve product packaging details forproduct identification.

Ubiquitous monitoring may optionally include the characteristic ofredundant monitoring. This may involve having redundant coverage frommultiple vantage points. For example, an item on a shelf may be visibleby two different cameras with adequate product identification resolutionand where the cameras view the item from different perspectives. In anenvironment like a grocery store this could mean 10-200 camerasdistributed per an aisle in some exemplary implementations.

While the system and method may have particular potential benefits to alarge environment, the system and method has potential benefits andapplications within small environments. Adaptability to differentenvironments is an exemplary benefit of the system and method.

The system and method may be used for any suitable video monitoringapplication. In particular, the system and method is for CV-basedcomputing applications, which may include automated self-checkout,inventory tracking, security surveillance, environmental event detectionand tracking, and/or other suitable applications.

Herein, automatic checkout is used as the main exemplary application ofthe system and method, but any suitable application may be used.Automatic checkout is primarily characterized by a system or method thatgenerates or maintains a virtual cart during the shopping process of acustomer (or group of customers) with the objective of knowing thepossessed items when a customer leaves a store or is ready for checkout.The system and method as described herein can be used to automaticallygenerate an itemized checkout list for a customer. An automatic checkoutsystem may additionally automatically charge an account of a customerfor the total of a shopping cart. The automatic checkout system couldalternatively be used to expedite entry of items for payment.

The system and method may alternatively be used to account for theremoval of a good by a customer such as in a library, a rental store, awarehouse, or any suitable item storage facility. The system and methodmay alternatively be used to permit or restrict access to locations, orto charge for such access. The system can be made to work for a widevariety of shopping environments such as grocery stores, conveniencestores, micro-commerce & unstaffed stores, bulk-item stores, pharmacies,bookstores, warehouses, malls, markets, and/or any suitable environmentthat promotes commerce or exchange of goods or services. In alternatecases, the system may be used in conjunction with law enforcement, orsimilar systems, to identify individuals who are not abiding by therules of the systems.

As one potential benefit, the system, and in particular the cameramodules, are preferably easily installed and activated. Physically, thecamera modules can be mounted using standard fixtures used in manystores. In some instances this may involve using existing storeinfrastructure such as those for lighting systems, and repurposing thosefor installation of the camera modules. In other instances, it mayinvolve standard device installation familiar to contractors andelectricians installing lighting systems and other common devices. Insome variations, the installation of the devices is substantiallysimilar to the structural challenges of installing traditional lightingfixtures.

From an electronics perspective, the camera modules may simply need tobe connected together with a set of cables. In some variations this mayinvolve a set of data and power cables. In some variations, this mayinvolve a single integrated cable for data and/or power connections suchas using a power-over-ethernet (PoE) cable or a custom cable.

The system and method additionally prioritize automatic configurationover customized configuration. The camera modules can be substantiallyself-initializing and the network of camera modules can self-organize tocoordinate environmental video monitoring. The self-calibration andconfiguration can minimize the amount of training for a workerinstalling the system. Such easy setup may be important to enabling anenvironment to come online with computer vision monitoring capabilitiesquickly. Additionally, the orientation and positioning of the cameraswithin the environment is made less necessarily precise. The form factorof the camera modules can simplify alignment and positioning of devicesfor desired ubiquitous monitoring. Camera alignment can be designed withspecifications within the normal tolerances of standard contractorinstallations. Additionally, self-configuration reduces reliance onspecialized technicians for bringing up and configuring the cameramodules individually. As opposed to maximizing the utility of eachcamera by planning and calibrating individual fields of view, the systemand method can address calibration by “saturating” an area with camerasand autonomously detecting the camera topology within the environment.The system can be used in creating a high-density imaging system withinan environment. In this way a worker installing a camera module can bealleviated of worrying about the exact orientation and positioning ofthe cameras.

As a related potential benefit, the system and method can additionallyhave easer maintenance. Various potential implementations of the systemand method can address various scenarios to simplify responding tosystem issues and/or updating the system. In some variations, componentredundancy designed into the camera modules can allow the monitoringnetwork to be resilient to individual failures by failing over toredundant data and/or power systems. This can increase uptime of thesystem. In another variation, an installed camera module (e.g., a cameramodule not functioning properly or an old version of a camera module)can be replaced by a new camera module 200. Such camera module updatescan be performed without having downtime on other camera modules.

As another potential benefit, the system and method can enable efficientconfiguration of camera compute modules. In one variation, fields ofview for a series of cameras can be coordinated such that theyefficiently cover a shelf, aisle, or other region of interest in amanner preconfigured through the arrangement and orientation of thecameras in the camera modules. In some variations, this may enable thenumber of cameras for an install to be significantly reduced.Additionally, some embodiments of the system and method can utilizecommodity imaging components to provide an economical solution tooutfitting an environment. The system and method can be operationalusing cheaper and more commodity components such as small format camerasensors (e.g., camera with a camera sensor sized: ¼ in., 1/3.2 in., ⅓in, 1/1.7 in., and/or other sizes).

As another potential benefit, the system and method can be a visuallyunobtrusive. With minimizing cabling and complex componentinstallations, the system may blend into the environment as a standardinfrastructure system that can go mostly unnoticed to people in theenvironment.

As another potential benefit, some embodiments may leverage CV-basedprocessing to address bandwidth data storage limitations. CV-basedprocessing can enhance information retention through dynamic informationprocessing that involves CV-based metadata extraction, media processing,and tiered storage solutions.

2. System

As shown in FIG. 1, a system for outfitting an environment withubiquitous video monitoring of a preferred embodiment can include amonitoring network 100 comprised of a set of camera modules 200 mountedand interconnected within an environment. Each camera module 200 of theset of camera modules preferably includes at least one camera 220 and atleast two connection ports 230. Within the monitoring network, the setof camera modules (or at least a subset of camera modules) seriallyconnect through the connection ports 230, wherein a physical connectionis established between connection ports 230 of network adjacent cameramodules. In this manner, the system can be used to form a connectedchain of camera modules 200 with shared network and/or powerconnections.

The system preferably enhances feasibility of implementing a ubiquitousvideo monitoring system. The system includes various factors thatbenefit the ease of installation as well as the reliable monitoring ofthe environment for serving a CV-driven application. Some variations mayinclude multiple cameras mounted in pre-configured arrangements in anenclosure, component and connection redundancy, hotswap capabilities,corrective optical systems, and/or other features. As discussed, thesystem is preferably used for CV-driven applications like automaticcheckout but can be used for other types of applications.

Preferably, the camera modules are devices that include multipleconnection ports 230 for receiving power and/or network connectivity andrelaying power and/or network connectivity to connected camera modules200. Camera modules 200 can be chained together by connecting with oneor more cables that provide data and/or power transport between thecamera modules 200. In one preferred implementation, the camera modules200 have an enclosure body 210 with an extended length profile that hasconnection ports 230 on opposing ends of the camera module 200 (e.g., afirst connection port 220 on a first side and a second connection port220 on a second side opposite the first side). The elongated lengthprofile functions to promote easier installation of cameras in apre-configured arrangement along a path (e.g., down an aisle in agrocery store). In one preferred embodiment, the camera modules 200 aresuspended fixtures interconnected by ethernet or PoE cables. The cameramodules 200 can contain one or more cameras 220 and possibly a set ofcomputational components to facilitate local processing of media. Insome cases, CV-based processing can be performed locally to mitigate theamount of media that is communicated across the monitoring network.

In some variations, the camera modules can include redundant data and/orpower connections for network resiliency. If a power connection fails, aredundant power connection can keep the camera module 200 powered.Similarly, redundant network connections can maintain networkconnectivity if one connection encounters an issue. Additionally, insome variations, a camera module 200 can include a hotswap controlsystem for removal and reinstallation of a camera module 200 such thatcamera modules 200 can be changed while the system maintains operation.

Monitoring Network

The monitoring network 100 functions to coordinate the collection ofvideo media data from across a number of camera modules 200 that areconnected as a camera module network. A CV-driven application willgenerally operate from data collected from one or more monitoringnetworks 100. Within an environment there may be a number of monitoringnetworks 100 (i.e., sub-networks). The camera modules 200 may beconnected in series within the monitoring network 100. The cameramodules 200 may alternatively include branches in the monitoring network100. In one variation, the monitoring network 100 can be comprised ofjust video camera modules. In another variation, the monitoring network100 may additionally include nodes comprised of other types of sensormodules/devices.

There may be a number of different ways to effectively configure themonitoring network 100 of such a system. In one preferred configuration,a monitoring network 100 within an environment is comprised of a set ofsub-networks formed by a serial connection of a set of camera modules200. A serially connected monitoring network (i.e., a daisy-chainnetwork configuration) preferably has each camera module 200 connectedto at least one other camera module 200 such that the string of cameramodules forms a chain. As shown in FIG. 2, from a network perspective,the series of camera modules may be the series connection of multiplenetwork nodes each having a number of camera connections.

In general, a first end (and/or terminal end) of the chain can includeconnections back to the rest of the system. A configuration couldadditionally include interconnection branches. For example, adaisy-chain of camera modules could split at some point so that thereare two or more branches. The number of camera modules included in aserial connection can be any suitable number. The number of camera railsin series could be just two, but more generally will be more than tenand could be any suitable number (e.g., 40 camera modules, 100 cameramodules, 1000 or more camera modules, or even more). As shown in FIG. 3,the set of serial sub-networks may all be connected back to one or morepoints for centralized compute and/or power supply.

In another configuration, the monitoring network 100 can be a starpattern, such that each camera module 200 is connected to a single,central component. The central component could be a primary cameramodule 200 (e.g., a hub camera module), but could alternatively be analternative central component interfacing with the different cameramodules.

Still another configuration is a loop or ring that is connected at twoor more ends. A combination of one or more of the above or alternativetopologies is also possible.

The system may additionally be compatible with external cameras, imagingdevices, other suitable sensor units, and/or output systems. In onevariation, camera modules 200 may additionally include compatibility tofacilitate a connection with an external camera such as an IP camera. Anexternal camera connection may alternatively be achieved through analternative channel such as through an external camera interface on alocal computing system. In one implementation, an external camera module200 can include similar operative components of a camera module 200 butcollects an external video stream of an external imaging system insteadof a video stream from an internal camera. The enclosure body 210 andform factor may be altered to accommodate interfacing with a variety ofcamera types. In this way, existing cameras and surveillance systems canbe retrofitted to integrate with the monitoring network 100 and thesystem.

The monitoring network 100 is preferably installed in a distributedmanner across an environment. The system is preferably adaptable forcustomizing the video monitoring so that focus is appropriately directedat subjects of interest. In many exemplary shopping environments, thesubjects of interest include the products as they are stored anddisplayed in the store and areas where customers, workers, or otheragents interact. As discussed herein, various approaches may be appliedin installing and configuring the camera modules 200 of the monitoringnetwork 100. A monitoring network 100 may employ a variety of suchconfiguration variations to address different monitoring scenarios inthe environment.

Camera Module

The camera module 200 of a preferred embodiment functions as an imagingdevice that is positioned within an environment. The camera module 200can function as one unit of the building blocks of the system inconstructing and laying out a monitoring network 100. A monitoringnetwork 100 is preferably comprised of a set of camera modules 200 thatcooperatively act to monitor and collect data from across theenvironment. More particularly, a monitoring network 100 can be one ormore sets of physically interconnected camera modules 200. A cameramodule 200 preferably includes at least an enclosure body 210, onecamera 220, and a set of connection ports 230. The camera module 200preferably facilitates image monitoring, but additionally facilitatesfixturing/mounting, network configuration, system integration, CV-basedprocessing and data management, and/or or other suitable aspects.

In at least one variation of the camera module 200, the enclosure body210 has an extended length profile, and the camera module 200 isconfigured with mounting mechanisms to be suspended horizontally from aceiling or elevated structure. There is preferably a plurality of cameramodules 200 used for any suitable instantiation of the system.

The camera modules 200 of a monitoring network 100 may be of a uniformtype and form, but a set of different camera module types can similarlybe used. In some cases, as part of a building block of the system, thereis preferably a set of camera module types that can be used to addressdifferent challenges when installing in an environment. Functionalityand internal elements such as the cameras and/or computationalcomponents are preferably substantially similar across different cameramodule types, but the enclosure body 210 may be customized toaccommodate different uses. In one implementation, the camera moduletypes may include different form factor variations and/or capabilityvariations.

The different form factors can accommodate different applications andscenarios where different shapes and/or sizes are more applicable. Someexemplary different camera module form factor types can includesuspended camera modules, mounted camera modules, flexible camera modulestrips, single-point camera modules, and/or any suitable type of cameramodule 200. Additionally, there may be different sizes or shapevariations of each of these.

Capability variations may enable different capabilities to be enabledand integrated at different portions of the monitoring network 100.Capability variations can include cameras with varying arrangement,orientations, camera counts, camera angle configurations, camera opticalsystems, enclosure body sizes and form factors, processing capabilities,and/or other suitable variations of a camera module 200. Camera moduletypes can have varieties in terms of lengths, capabilities, and/or otherproperties.

In one variation, capability variations can be two camera module types:a normal camera module 200 and a hub, central, master, or primary cameramodule 200. The primary camera module may include additional computingresources to facilitate processing data of the system. As the additionalcomputing resources may be needed in limited supply and contribute tocost, they may be provided as specialized camera modules that can beused as needed in a monitoring network.

The camera modules 200 are preferably interconnected by wired conductiveconnections that interface with the camera modules 200 through at leastone of the connection ports 230. The connection ports 230 can connectpower and/or a data connection (e.g., network connectivity). Wirelesscommunication may alternatively be used for one or both of power and/ordata connectivity. In one example, multiple sub-networks of cameramodules can be integrated as shown in FIGS. 1 and 3.

Additionally, to facilitate easy configuration, camera modules 200 maybe designed with redundant or optional components that can be enabledfor different monitoring network 100 configurations. For example, whilewired networking is preferably used. The camera modules may additionallyinclude a Wi-Fi module or other wireless communication module so that acamera module 200 or a subnetwork of camera modules could communicatewirelessly with the system. Wireless power delivery may additionally oralternatively be used.

A camera module 200 preferably includes at least two connection ports230 such that the camera modules may be connected in series. With anenclosure body 210 with an elongated form a first connection port 230can be positioned on one end and the second connection port 230 can bepositioned on the other end. The connection ports 230 may alternativelybe positioned in any suitable arrangement (e.g., side-by-side). Somevariations can include three or more connection ports such that amonitoring network 100 can include multiple branches. For example, oneend of a camera module 200 can include two connection ports 230 suchthat a series of camera modules 210 can branch or merge at that end ofthe camera module 200.

As discussed, the camera modules 200 are part of a modular set ofcomponents available through the system, which can be used incustomizing configuration and arrangement of the camera modules 200 tobe customized to a particular environment and/or use case. Differentenvironments and use cases can have different requirements. Environmentsmay have various static or changing visibility expectations that canimpact the spacing and arrangement of camera modules. Similarly, the usecase or objective of the system can have a similar impact onarrangement. For example, when used in a store or a warehouse, thelayout of inventory and shelving can be a factor when configuring cameramodules. As another example, using the system for detailed inventory andcustomer interactions may have different coverage requirements.

In particular for some variations, the system can be used for monitoringitems on the shelves and interactions with those items. The installationconfiguration of one or more camera modules within the monitoringnetwork can be designed to specifically address challenges of monitoringinventory and inventory interactions. In some cases, the camera module200 design and features may be customized for specific installationplans.

As shown in FIGS. 4A and 4B, a dual-angled center camera module 200 canbe positioned in the upper region or above the center region of an aisleto monitor both sides of the aisle. Some stores have existinginfrastructure for suspending lights down the middle of aisles. In somevariations of the camera module 200, a camera module 200 with lightingfixture integration could be used so that a) the camera modules may bemounted to existing infrastructure by replacing existing lightingfixtures, and/or b) the system can provide lighting to the environmentin those regions.

As shown in FIG. 5, another installation configuration could have atleast two rows of connected camera modules running down either side ofan aisle in a store. In this variation, installing the camera modules200 above and preferably in front of the face of a shelf (e.g., 5 inchesto 10 inches in front of a shelf face) may enable the camera module 200to both monitor items on the shelf opposite the aisle and monitor whenand approximately where customers interact with the shelf face closestto the camera module 200. In one implementation, a single camera canhave a field of view incident on a first shelf face and the oppositeedge of the field of view incident on the upper region of the oppositeshelf face as shown in FIG. 5. In another implementation, a dual-anglecamera module 200 may have one camera directed substantially in a firstorientation (e.g., downward) where a first shelf face is within thefield of view and a second camera directed in a second orientation(e.g., outward diagonally) where the second/opposite shelf face iswithin the field of view of the second camera.

Other installation configurations may also be used. In some variations,the camera modules may be installed so that they run perpendicular to ashelf face or a target region. In another variation, the camera modulesmay be installed so that the camera module 200 collects image dataspanning multiple aisles as shown in FIG. 6.

Camera Module Body

The enclosure body 210 of a preferred embodiment functions as a framefor mounting the cameras 220, mounting and protecting computationalcomponents of the camera module 200, and/or housing other elements ofthe camera module 200. The structural body is preferably a solidencasement that contains the computing components. The enclosure body210 can include a structural body with at least one defined cameramount. A preferred variation of camera module 200 can include aplurality of cameras 220 wherein the plurality of cameras 220 aremounted with a spaced arrangement across the enclosure body 210 indifferent defined camera mounts. The enclosure body 210 can additionallyinclude one or more mounting mechanisms that function to facilitatepositioning, fixturing, or otherwise physically coupling the cameramodule 200 in place.

The structural body is preferably enclosed, but alternative form factorssuch as an open tray could also be used. The structural body can be madeof a material such as plastic, aluminum, steel, a composite, or anysuitable material. In one preferred implementation, the structural bodyis fiberglass, which could minimize signal interference. A structuralbody can be a single piece design, but the structural body canalternatively include multiple interconnecting components.

The structural body is preferably a static element with one or moredefined cavities for holding the camera and computational components.Preferably the camera modules 200 support “daisy chaining” or“stringing” camera modules across an environment. Different form factorsof the structural body can be used to accommodate installation indifferent types of sites. Two preferred variations include a railstructural body, tile structured body, and/or a compact structural body.

In a rail structural body variation, the structural body has extendedlength profile, wherein the magnitude along one dimension of thestructural body is significantly greater than the other two dimensions(e.g., the length is significantly greater than the width or depth).Described another way, an enclosure body 210 can have a rail-shaped asshown in FIG. 7, tube-shaped enclosure as shown in FIG. 8, or anysuitable shape. The shape is preferably straight but curved, arced, orangled enclosures could additionally have an extended length profile. Anelongated form factor may function to facilitate arranging cameras alonga long run in the environment. For example, for a given length of space,an elongated form factor can have a lower number of units-per-lengththat need installation thereby reducing labor for installation.

A rail structural body variation can come in different lengths. Someexemplary lengths may include lengths that are somewhere between 3-10feet. In some instances, a set of different lengths may be used: a 2-3foot enclosure body, a 5-6 foot enclosure body, and a 9-12 footenclosure body. Any suitable length may alternatively be used.

In a tile structural body variation, the structural body may be designedfor distributing camera modules 200 by “tiling” or “spreading” cameramodules as shown in FIG. 9A. A tile structural body can be an enclosurebody with a form factor that has at least one substantially planar faceand as such can be a plate or sheet. The planar face will generally bein the shape of a square or rectangle but any suitable shape may beused. The planar face preferably is used to expose the cameras while theopposite side is preferably used to expose connector ports andoptionally house the components of the camera module. A structural bodyvariation may function to make the camera modules 200 compatible withceiling tile infrastructure as shown in FIGS. 9A and 9B. The cameramodules can be installed by replacing existing ceiling tiles with a tilevariation of the camera module (i.e., a camera module tile). A cameramodule tile can have a structural surface shaped two feet by two feet,two feet by four feet, or have any suitable ceiling tile compatiblesize. The camera module tiles are preferably compatible for drop ininstallation to t-bar or suspended ceiling fixtures used for dropceilings as shown in FIG. 9A. In a related variation, the structuralbody can be designed for adding on to or retrofitting an existingceiling tile or material. In this variation, the camera module may bedesigned such that the cameras 220 project through the ceiling tile ormaterial such that the cameras view the target region. Some or allhousing of camera module components may be designed to be positionedabove the ceiling tile out of site as shown in 9B.

The rail structural body variation and the tile structural bodyvariation the camera modules may be positioned in a regular fashionwherein the arrangement of the cameras can be substantially periodic andregular across multiple interconnected camera modules. As shown in FIG.10, a five and half foot camera module rail could be interconnected withsix inch spacing between camera module rails such that pairs of camerasin the monitoring network have three foot spacing.

In a single-point camera module, the structural body may be designed tobe compact such that the camera modules 200 can be installed at distinctpoints. The single-point camera module may include a single camera mountbut more preferably will include multiple camera mounts directed indifferent directions as shown in FIG. 11. In one implementation, asingle-point camera module can be a suspended camera module 200.However, the mounting mechanism of the single-point camera module can beany suitable mechanism to fasten or position the camera module 200. Asingle-point camera module may include the variations described hereinsuch that it can connect to one or more camera modules through a wiredconnection. However, a single-point camera module in particular mayalternatively make use of a wireless communication channel such thatonly power is delivered through a wired connection. A single-point bodymay be designed for more point-based installations where theinterconnections of camera modules are not substantially interconnectedin close proximity as in FIG. 10.

Herein, the rail structural body variation is primarily used as anexample but any suitable variation could alternatively be used.

The varying form factors may be mounted in various manners within theenvironment. An enclosure mounting mechanism is preferably a fastenermechanism or design feature of the enclosure body 210 that promotespositioning, attaching, or otherwise physically coupling a camera module200 to an outside structure. The nature of the mounting mechanism canvary depending on the camera module type. Selected variations aredescribed more below.

As shown in FIG. 8, a suspended camera module 200 is a form factor thatis configured to be suspended from a higher structure such as a ceiling.A suspended camera module 200 is preferably used in monitoring anenvironment from the ceiling region, and it may be straightforwardsolution for commercial spaces that already leverage suspended fixturessuch as lights and signage.

The mounting mechanism of a suspended camera module 200 preferablyaccommodates one or more cable connection points. In one implementation,the enclosure body 210 can include two threaded lamp tube fasteningpoints along the length of the enclosure body 210 and on opposing sidessuch that two cables can be used to horizontally suspend the cameramodule 200 from the ceiling. In another exemplary implementation, amounting mechanism (e.g., protruding bolts) may be designed for mountingto a supplementary structural element like a unistrut or cable fixturingmechanism.

In another variation, a mounted camera module 200 is preferablyconfigured to be rigidly fastened or attached to another rigidstructure. For example, camera modules may be mounted directly to ashelf as shown in FIG. 12 so as to horizontally extend across an aisleof adjacent shelves. The mounted camera module 200 can include many ofthe suspended camera mount design considerations. However, as a mountedcamera module 200 may be in closer proximity to users, mounted cameramodules preferably include design considerations to more rigorouslyprotect from tampering and abuse. The mounting mechanism can be definedto include bolt through-holes so that the mounted camera module 200 canbe securely fastened to another structure. With similar robustnessconsiderations, a mounted camera module 200 may additionally include aprotective camera cover so that wear-and-tear near the camera mountareas can be addressed by replacing the camera cover as opposed tochanging the camera module 200 or the camera.

In another variation, the system may include a bracing structure thatacts a rigid structure to suspend or otherwise position a mounted cameramodule 200. As shown in FIG. 13, a shelf-adapter structure may attach tothe top shelf of a shelving unit and suspend a lever outward to acorrect mounting position. This may function to simplify installationalong aisles and to promote controlled positioning of a camera module200 relative to the shelf face.

The camera mounts function as locations where a camera is mounted.

In one variation, the camera mounts can be defined cavities or recessesof the structural body such that cameras can have an outward view of theenvironment. In one variation, the camera mounts may additionallyfacilitate connected camera units such that in place of fixed cameras,interchangeable camera units can be connected or plugged into a portaccessible through the camera mount and held in the camera mount.Interchangeable camera units may enable camera customization andupgrades. The camera mounts may be statically positioned. Alternatively,the camera mounts may be repositioned either through physicalmanipulation or through a controlled camera actuator system. Thecustomization of camera angle when mounted may be through an articulatedmechanism (e.g., cameras can be moved into position) or through a staticfixturing approach (e.g., cameras are mounted in one of a range ofpossible positions.) In one exemplary implementation, a segment of theenclosure body 210 with extended length profile can include a cameramount section that can be rotated about the central axis of theenclosure body 210 to rotate the angle of the camera. In anotherimplementation, camera angle may be customized during final assembly ofthe camera module 200.

Camera Module Variations

In one variation, a camera module 200 may include multiplesub-assemblies that can be connected to form a complete camera module200. In one example, a suspended camera module 200 may be formed byconnecting two partial camera module halves through a mechanical andwired connector. Each camera module unit may include the necessarycomponents. Alternatively, some camera modules may omit or containparticular components such that camera module units may require beingconnected in a combination such that particular component requirementsare fulfilled. For example, a special processing camera module unit maybe required to be connected to any segment of a camera module assemblyfor every six active cameras.

As another modular variation, each camera module 200 may includeconnectors on one or both ends such that any suitable number of camerasub-system units 202 (i.e., camera module units) can be combined to forma camera module assembly. A camera sub-system unit 202 is preferably acomputing device such as a PCB board or enclosed computing device. Astructural body can have internal fixtures to mount varying numbers ofcamera sub-system units 202 as shown in FIGS. 14A and 14B. In theexample of FIGS. 14A and 14B, the different sub-system units 202 eachsupport some number of cameras and thus the number of cameras in acamera module 200 can be adjusted by selecting the number of enclosedsubsystem units 202. In this way, camera modules may be flexiblycustomized and repurposed for different environment challenges.

For structural stability, the structural body can be a substantiallyrigid material such as steel, fiberglass, aluminum or plastic, but anysuitable material may be used. In one variation, the enclosure body 210can be a length of tubing, bar, or other form of self-contained rail asshown in FIG. 15A, but may alternatively be an open trough.Computational components are preferably contained within a definedrecess or hole of the enclosure body 210. The cross section of thestructural body preferably includes subsections or structures thatextend upward, which functions to act as containing walls and to providestructural support during suspension. As shown in FIGS. 15A-15E, variousprofiles may be used. A trough design may additionally include a covercomponent to at least temporarily seal contained components within theenclosure body 210 as shown in FIGS. 15B and 15C. Downward facing wallsof the structural body can be angled so as to position cameras at anintended angle. In one implementation, a right angled extruded bar canhave camera mounts oriented on each face so as to monitor differentdirections as shown in FIG. 15B. In one variation, the angle may beadjustable. In one variation, the structural body may have a sectionwith multiple facets and/or camera mount options as shown in FIG. 15C sothat positioning of the camera can be selected during manufacturingand/or installation. The cameras may be mounted ‘proud’ as shown in FIG.15D, or recessed slightly as shown in FIG. 15E so that they aremechanically protected while still having a full optical view of theirsurroundings.

As one alternative variation, a flexible camera module strip canfacilitate a camera module 200 being adhered or otherwise attached to asurface. As opposed to a structural body made of a rigid material, thestructural body of flexible camera module strip can be bendable alongsubstantial portions of its geometry as shown in FIG. 16. Some portionsmay still be made rigid to protect rigid internal components. In onepreferred embodiment, the flexible camera module strip includes anadhesive backing opposite the side of the camera mounts such that theflexible camera module strip can be installed by simply applying it to asurface.

Cameras

The cameras 220 of a preferred embodiment function to collect imagedata. The image data is preferably a video stream but couldalternatively be periodically collected photographs. The video streammay or may not include audio recorded by a microphone. The cameras 220may alternatively be used to collect other forms of image data. A camera220 of the system may collect any suitable combination of visual,infrared, depth-based, lidar, radar, sonar, or other types of imagery.

The set of cameras used within the system or a single camera module canhave varying resolutions, fields of view, aperture settings, framerates, capabilities, or other features. The cameras 220 are preferablystatically mounted in a camera mount of a camera module 200.Alternatively, the cameras 220 and/or other camera mounts may beactuated such that a camera can be redirected during use.

An individual camera module 200 preferably includes at least one camera220, but may alternatively include multiple cameras 220. Multiplecameras 220 may be mounted at different locations on a camera module200. The cameras 220 can be mounted to have distinct positions and/ordirections such that the cameras 220 have a configured arrangement wherethe arrangement characterizes spacing and relative orientation.Preferably, multiple cameras 220 are mounted so as to have a distinctfield of view, which may be overlapping or non-overlapping with a fieldof view of another camera 220 in the monitoring network 100 or thecamera module 200. In one implementation, multiple cameras 220 may bemounted at opposing ends of an extended length camera module 200. Asubset of cameras 220 may alternatively be mounted so as to capturesubstantially similar fields of view but using different imagingtechnologies.

There can be a variety of camera module 200 types with varyingconfiguration just based on options in selection of angular orientationas part of the camera arrangement. In a first variation, a first subsetof cameras are mounted with a first angle orientation. As part of asingle-angle variation, all cameras of a module may have the sameangular orientation wherein the image planes that are substantiallyparallel. In multi angle variation, a first subset of cameras aremounted with a first angle orientation and at least a second subset ofcameras are mounted with a second angle orientation. A dual anglevariation will have two subsets of cameras with different angularorientations. For example, a group of cameras may be directed in onedirection and another group directed in a second direction as shown inthe exemplary application of FIG. 4A. A Tri-angular variation may beanother common variation, wherein there may be two diagonal angleorientations for two subsets of cameras and one downward facing angleorientation of a subset of cameras. Any number of angular orientationsmay be used.

The cameras are preferably angled so as to be directed at a targetregion. In one common scenario the target region is a shelf or morespecifically, the shelf face. In many situations, the camera module 200will be mounted above and in front of the shelf face. Shelves havevarying heights depending on the store and type of goods. In general,the camera is mounted above the region of the shelf face of interest.The camera module may be offset from the shelf face horizontally by afew inches (e.g., five inches) to several feet (e.g., 15 feet). Ingeneral, the camera will be somewhere between three to eight feetdisplaced from the shelf face in a horizontal direction. The cameraswill often be elevated above the top surface of the shelf with verticaloffset of 0 feet (i.e., level) to 10 feet. Though the camera may bebelow the top in some cases. In one situation, a subset of cameras of acamera module may be mounted with an angle orientation configured tocapture a first shelf face

In another situation, the camera may be mounted between two opposingshelf faces. The angle orientation of camera mountings is preferablyconfigured to target the shelf faces. More specifically, a subset of thecameras are preferably mounted with an angle orientation that isconfigured to capture a first shelf face of a first shelf and a secondshelf face of a second shelf when the camera module is mounted above andbetween the first and second shelves as shown in FIG. 5. The first andsecond shelf will generally oppose each other. In this way, a singlecamera can have two shelves in the field of view. The first shelf may becaptured to identify products and/or detect item interactions by acustomer. The second shelf may be captured for similar reasons, but mayadditionally be monitored so as to detect when an interaction eventoccurs with the shelf.

In one preferred variation, cameras 220 may be mounted as camera pairs,wherein a subset of cameras are paired in a depth perceptionarrangement, which functions to enable depth calculation to beperformed. Depth sensing and estimation could additionally oralternatively be performed using alternative techniques including singlecamera depth prediction. The depth perception arrangement of camerapairs is preferably spacing between one and ten centimeters apart, butany suitable spacing may be used. Multiple camera pairs are preferablyarranged along the enclosure body. For example, a six foot camera railcould have two camera pairs spaced three feet apart directed at a firstorientation (referred to in this example as first-angle cameras. In amulti-angled camera variation, two additional camera pairs (referred toin this example as second-angle cameras) may each be positioned alongside one of the first-angle camera pairs such that the second-anglecamera pairs are similarly spaced three feet apart. The second-anglecameras are oriented with a second angle. N-angle cameras may be used.

The cameras 220 can additionally include an optical system 320 whichfunctions to better target monitored objects. While the cameras 220 mayprovide general surveillance, cameras 220 in CV-driven applications mayhave particular benefits when customized to collecting image data ofparticular regions in an environment. For example, in an automaticcheckout usage scenario, the suspended camera modules will preferably beable to reliably provide image data on shelved items. When the cameramodule 200 is suspended, the plane of the shelves will generally beaskew from the field of view of the camera's 220, which could result ina keystoning distortion effect of the shelf and products. Additionally,the focus plan of a camera will generally not be parallel to the shelfand portions of the shelf may be out of focus despite focusing thecamera on some region. In some instances, the degree of focus and out offocus of the items may not be an issue.

A corrective optical system 320 can facilitate correcting fororientation misalignment of the camera imaging plane and a subjectplane, which can mitigate distortion and/or improve focus for regions ofinterest. The optical system 320 preferably optically shifts and/ortilts the focus plane of a camera to counteract a portion of distortionand/or focus when imaging a target subject. CLAIM-16A corrective tiltshift (e.g., Scheimpflug) optical system can create a wedge shaped depthof field that can be aligned with the subject plane (e.g., the frontplane of a shelf). In some variations, the corrective optical system mayapply an optical shift, which can correct distortion. Additionally oralternatively, the optical system may apply a tilt to alter the focusplane to better align with the target region of a shelf face. This canbe used to have products positioned from bottom-to-top in focus. Asshown in FIG. 17, a lens can be positioned non-parallel or tiltedrelative to a camera imaging plane to correct imaging for shelf viewing.The FIG. 17 is not to scale so as to better illustrate the basicprinciples of one optional optical system. A compound lens or otheroptical setup can additionally be used. Additionally or alternativelyother optical system variations such as a lenticular or alternative lensused for a light field camera or a plenoptic camera, use of multipledirected cameras, or other suitable cameras and/or optic systems couldbe used in adapting the collection of image data for a particulartarget.

Many variations of the optical system may be applied which is describedin more detail in the section below.

Supplemental Components

A camera module 200 can additionally include other supplementarycomponents used in offering additional or enhanced sensing orfunctionality. Supplementary components may include microphones,speakers, area lighting, projectors, communication modules, positioningsystem modules, and/or other suitable components.

In one variation, the camera module 200 can include microphones suchthat a distribute audio sensing array can be created. Audio sensing canbe used in identifying, locating, and collecting audio input fromdifferent locations. For example, a monitoring network 100 withmicrophones can triangulate sounds to determine location within theenvironment. This can be used to facilitate CV-based tracking. Thiscould alternatively be used in enabling audio-based interactions withthe system. In one variation, the microphone array provided through themonitoring network may be used to facilitate in-store audio-interfaces.For example, a customer could issue audio commands from any place in thestore, this could be synchronized with the CV-driven application whichmay be used to associate a detected audio command with a user entity oraccount issuing that command. In one implementation, the microphonearray may be used in differentially locating, processing, modifying, andresponding to audio sources as discussed in published U.S. patentapplication Ser. No. 15/717,753, filed 27 Sep. 2017, which is herebyincorporated in its entirety by this reference.

In another variation, the camera module 200 can include integratedspeakers, which can function to enable audio output. In oneimplementation, this may be used to simply play audio across anenvironment. The speakers are preferably individually controllableacross the monitoring network, and targeted audio could be played atdifferent regions. In the automatic shopping experience, this can beused in providing augmented audio experiences as a shopper is trackedthrough a store. The speakers could additionally be used as a humancomputer interface output when configuring or maintaining the device.For example, a camera module 200 could be set to play an audio signalwhen the camera module 200 enters an error state.

In another variation, the camera module 200 can include a lightingsystem, which functions to at provide general lighting. The lightingsystem could include integrated lights. Integrated lights could be LEDlights or other suitable light sources that are built into the cameramodule 200. The lighting system could alternatively be a set of lightingfixtures such that external lights could be connected to. In somevariations, a lighting fixture is designed to power traditional lightssuch as fluorescent lights, LED lights, incandescent, CFL lights, andthe like could be installed and powered by the system. An integratedlighting system can enable the infrastructure of a store to be minimizedby not needing to set up separate lighting and camera systems. In someinstances, the structural infrastructure used to support and optionallypower existing lights can be repurposed for fixturing and/or poweringthe camera modules 200. When the integrated lighting includes LEDlights, installing the camera modules 200 and monitoring network mayserve to upgrade environment lighting as well as adding advancedmonitoring and CV-driven applications. The enclosure body 210 can beadjusted to support integration of the lighting system as shown in FIG.19A.

Alternatively, a camera module 200 could have a form factor such thatthey can be installed into pre-existing lighting fixtures. For example,a “fluorescent tube” camera module form factor shown in FIG. 19C or a“light bulb” form factor as shown in FIG. 19B could be used such thatcamera modules could be fixtured in place and powered by being insertedinto a lighting fixture. Integration of a lighting system can enable thecamera module 200 to still provide light like a normal light, but to beenhanced with the sensing and computational capabilities of the cameramodule 200. These variations may communicate wirelessly or have anetwork connection port that is exposed when inserted in the electricalfixture.

The lighting could additionally be dynamically and individuallycontrolled by the system. The on/off state, brightness, colors,directionality, and/or other properties could be individuallycontrolled. In combination with the CV-driven capabilities offeredthrough the system, lighting could automatically be adjusted based onobserved objects in the environment. In one example application, duringa power saving mode used within a store at night, the lights couldautomatically turn on and off or dim based on the location of the people(e.g., workers) present in the environment.

In another variation, the camera module 200 can include a projectorsystem, which functions to project structured lighting. Images can beprojected at different locations using a projector system. The camerasof the system are preferably used as a CV-driven sensing input that canbe used in various applications, and a projector system can be used as asystem output to compliment the computational input from the cameras.The projectors can be individually controlled and can similarly be usedin combination with a CV-driven application.

In another variation, the camera module 200 can include a communicationmodule, which functions to facilitate a communication or data networkwithin the environment. The communication module could be a WiFi routerused to provide wireless internet within the environment. Thecommunication module could alternatively be a Bluetooth module used forBluetooth beaconing or other applications. Any suitable type ofcommunication module could be integrated into the camera modules or aportion of the modules to provide a wireless communication channelwithin the environment.

In a related variation, the camera modules may include a positioningsystem, which functions to act as a mechanism for local positioningwithin the environment. For example, an RF-based positioning systemcould be used to track RFID tags in the environment.

Other alternative sensors and devices could additionally be includedsuch as environmental condition sensors (e.g., smoke alarms, CO₂monitors, temperature, humidity, air quality, etc.) or other componentsfor different functionality.

In one variation, the components of the camera module 200 can bearchitected and designed for operational redundancy. In thedaisy-chaining variation, the system can preferably maintain operationof the camera modules 200 even if one camera module 200 fails.Computational and communication redundancies can enable a sequence ofinterconnected camera modules to keep operating even when one segmentfails. In one preferred implementation, a camera module 200 preferablyhas redundant power and/or redundant network connectivity. A powercontrol system may manage redundant power. A network switch or hub maycoordinate redundant network connectivity.

In one implementation, a camera module 200 can include a connector oneach end for daisy-chaining, a second connector on each end for formingT- and X-shapes, and a second set of each pair of connectors forredundancy (now 8 connectors in all). The pair of pairs can each becollected into a network (e.g., net-A and net-B as shown in FIG. 20).The four connections that make up net-A are connected to a common hub,as are the connections that make up net-B. Hub-A and hub-B are then,in-turn connected to hub-C, which connects to the CPU. The CPU connectsto the array of cameras. This configuration can be fully robust againstthe failure of any single connector or of either hub-A or hub-B.Further, if hub-C or the CPU fail, connectivity to the other cameramodules in the system are preserved.

In an alternative implementation that is further enhanced, a second CPUis added, and each of the two CPUs is given authority over managingone-half of the camera elements as shown in FIG. 21. In thisimplementation, if either hub or CPU fails, half of the cameras willstop operation. The configuration of the camera module 200 and/ornetwork preferably has a significant enough camera density that thecamera coverage redundancy is sufficient to handle some camera failures.This second implementation can be further augmented by adding aconnection CPU-B & hub-A; and CPU-A & hub-B.

In addition, a camera module 200 can include a hotswap control systemsuch that a camera module 200 may be safely disconnected or connectedfrom the system while the system is live and operating. The hotswapcontrol system can additionally offer continuous protection from shortcircuits and overcurrent faults. Hotswap control system can enablehardware updates and fixes to the system to be performed withoutrequiring the system to be powered down. As an example, a camera module200 may encounter an issue where it needs to be replaced. A maintenanceworker can rewire network and/or power connections around the cameramodule (e.g., swapping redundant power or network connections to asubsequent camera module or to a planned replacement), and then removethe camera module causing the issue. A replacement camera module canthen be primed for connection by rewiring the network and/or powerconnections, and then swapped into the monitoring network 100.

Computational Components

The camera module 200 preferably includes a set of computationalcomponents used in performing local processing, managing state of thecamera module 200 and/or interfacing with the monitoring network, othersystem elements, or remote resources.

The computational components preferably include a subset of componentsfor processing. The processing components can manage operating state andother operations of the camera modules 200. In one preferred variation,the camera module will include at least one processing unit configuredto perform local processing of collected video data. As one aspect, theprocessing components preferably function to transcode or transformimage data from a camera to a format for digital communication withother resources of the system. The processing components can include ageneral processing unit, a specialized processing unit such as agraphical processing unit (GPU), a dedicated computer vision or deeplearning processor, and/or any suitable processing component.

In one variation, the processing performed at the processing componentsmay include transforming raw image data to a compressed media formatsuch as a MPEG video format. In another variation, image data may, attimes, be transformed from raw image data to a metadata representationwherein a CV analysis extracts information from the image data. Themetadata and/or the digital media format may be communicated to othercamera modules and/or system components.

In one variation, the camera modules 200 may be configured toselectively communicate video data based on results of local processingof video data. Media transformation may be dynamically set according to“information quality” of the metadata. The “media quality” of theresulting media format during transformation may be indirectly set basedon the “information quality”. For example, when the media is transformedinto a metadata representation with a high level of confidence, themedia format could be reduced to save data storage or bandwidth. As anexample, video media may not be stored or communicated if no motion isdetected, if no people are present, and/or if other suitable conditionsare met.

A camera module 200 may include a processing unit, GPU or othercomputational elements for each camera, but they may alternatively beshared across cameras.

The computational components can additionally include a communicationcontrol system, which can be a networking hub or switch. The networkingswitch functions to facilitate network configuration and/orcommunication in and out of the camera hub. Communication is preferablyperformed over a wired connection to a connection port of the cameramodule 200, but wireless communication may additionally or alternativelybe used. Internal or external DHCP can be used to distribute networkconfiguration to the various components of the monitoring network 100.The networking switch is preferably a networking router that isconfigured to perform DHCP internally when more than one processor ispresent, wherein the assignment of IP addresses can be managedinternally by a master router of the system, such that the camerarouters can be self-assembling from a networking perspective. As shownin FIG. 2, the cameras of the monitoring network may be grouped across aseries of such network switches. In one variation, the network switchescan implement a form of Spanning tree protocol (STP) or more preferablyrapid spanning tree protocol (RSTP). Other suitable network protocolscould similarly be implemented.

The computational components of a camera module 200 can additionallyinclude an onboard data storage/memory solution, which can be used instoring media and/or data. The data storage solution could be a harddrive, a solid-state drive, and/or any suitable type of data storagesolution. The data storage solution can be used as a temporarily stashof media and/or data from the camera hub. Data storage may beprioritized for potential long-term relevance.

The computational components can include other common components such asonboard memory, data ports, or other components. The camera module 200may additionally include various user interface elements as describedbelow.

User Interface Elements

The camera modules 200 are preferably designed to simplifyconfiguration, and during normal usage, direct interaction with a cameramodule 200 may not be needed. The camera module 200 may, however,include user interface elements to either output information or enabledirect control of the camera module 200. User interface elements mayinclude basic elements such as buttons or user physical device inputelements, displays, indicator lights, speakers for audio signals, andthe like.

In one preferred variation, the camera module 200 may include a controlchannel mechanism that can enable connected control of a camera module200 through a secondary device. A control channel mechanism can functionto alleviate camera modules from being built with extra components forthe rare situations where manual control is required. Control andconfiguration can preferably be performed through the network connectionof a camera module 200 as well. A personal computer, a smart phone,tablet, or the like may be able to connect to the camera module 200 andused as a medium for interfacing with an individual camera module 200. Aphysical connection may be used in the situation where networkconnectivity to the camera module 200 is not working or is notavailable. A user can directly connect to the camera module 200 tocollect diagnostics, access data, update configuration, control thedevice, and/or otherwise interact with the camera module 200. A controlchannel connection can be done through a cable such as a wired connector(e.g., a serial port, RJ-45, or a USB connection) or a wirelessconnection such as Bluetooth or a Wi-Fi network established by thecamera module 200, or any suitable communication channel. In onevariation, the camera of the camera module 200 may be used as data inputchannel to the camera module 200. For example, a smart phone app mayenable configurations to be set and then an encoded message can bevisually communicated to the camera module 200 through the display or alight of the camera. Output from the camera module 200 may be indicatedthrough indicator lights or other suitable feedback mechanisms.

Connection Port and Connections

The connection ports 230 functions to facilitate ease of wiring of thecamera modules by utilizing serial connections of a sequence of cameramodules 200. Preferably, there are two connection ports which functionsto allow the camera module to serve as a power and/or networkconnectivity relay wherein input power or network connectivity isreceived from one connection port 230, utilized by the camera module,and relayed to at least a second connection port 230. This may be usedfor serial connections of camera modules 200 as shown in FIG. 22. Theremay additionally be three, four, or any suitable number of connectionports 230 where the additional connection ports 230 can be used tobranch delivery of power and/or network connectivity (e.g., creating Tor X branches) as shown in FIG. 23. Different connection ports 230 of acamera module 200 are preferably substantially the same. The connectionports 230 may also be non-directional where any connection port 230 maybe used as an input or output (i.e., supply or relay) of power and/ornetwork connectivity. For example, the direction a camera module 200 isconnected may make no difference. Alternatively, directionality may bebuilt into the design of the camera module.

As one example, instead of dealing with running power and networkinglines to 100-1000 different devices to monitor a store (which may beaccompanied with a high level of installation complexity), one totwenty-five “chains” of camera modules may be used to monitor a store.The set of camera modules that are part of the monitoring network 100are preferably interconnected through wired connections. The wiredconnection is preferably used in electrically coupling camera modulesfor delivering electrical power and/or data communication. The cameramodules 200 preferably include at least two connection ports 230 suchthat a chain of camera modules can be connected. Each connection port230 may be exposed on one side of a camera module 200 as shown in theend view of a camera module 200 in FIG. 24. A first connection port 230preferably functions to provide power and/or network connectivity, whilea second connection port 230 functions to pass along connectivity topower and/or network access to subsequent camera modules. The cameramodule 200 may be designed such that power and/or network connectionsare independent of connection port 230. For example, a camera modulerail could be connected in a series of camera modules with any suitableorientation. In some variations, the camera modules 200 may include atleast three connection ports 230 such that a chain of camera modules canbranch off in different directions.

In one preferred variation, each connection port 230 includes both apower connection port 232 and a communication connection port 234. Thepower connection port 232 functions to form a connected power deliverynetwork across a network of connected camera modules. The communicationconnection ports 234 function to form a communication network across aset of cameras mounted within the environment.

As one alternative variation, a connection port 230 may include a powerconnection port 232, wherein communication may be facilitated wirelesslyor in another manner. As another alternative variation, a connectionport 230 may include only a communication connection port 234, whereinpower may be facilitated in another manner (e.g., having a separatepower cord). In the described variations, there may be redundant orsecondary power connection ports and/or communication connection ports.Redundant connections may not be inherently redundant or failover butmay be designed where port usage is equally prioritized or selected inan alternative manner. Alternatively, one of the connection ports may bea primary one used unless some event triggers use of thesecondary/redundant one. As shown in FIG. 22, one preferred variationcan have two power connection ports 232 that connect a DC powerconnection and two communication connection ports 134 that connect twonetwork cable connections.

Multiple distinct cables may be connected to a connection port 230.Alternatively, an integrated cable may be used to connect to aconnection port 230 and thereby one or more power connection port(s)and/or communication connection port(s). In one implementation, thewired connection can be a power over ethernet (PoE) port. The wiredconnection could alternatively be a USB port (e.g., for a USB PowerDelivery connection) or any suitable wired connector. In anothervariation, a custom cable type could be used that bundles power andnetwork connectivity.

For the power connection port 232, the camera module 200 preferablyincludes a power control system that functions to regulate or otherwisemanage power supplied to the components of the camera module. The powercontrol system further functions to relay or pass power supply betweenconnection ports. For example, a 60 VDC power supply received through afirst power connection port 232 is relayed through the power controlsystem (e.g., power management circuitry) to at least a second powerconnection port 232. Another camera module connected to the second powerconnection port 232 can receive power through that connection. The powercontrol system will preferably step down voltage to one or more desiredvoltage levels to drive components of the camera module or morespecifically components of a camera sub-system unit 202. Additionally,the camera module 200 or a camera sub-system unit 202 can be designed tohandle sufficient current loads in the event that a camera module is atthe beginning region of a chain camera modules 200. For example, thepower system of the camera module and/or a camera sub-system unit 202can include a high-current capacity channel that carries current betweenconnection ports 230.

With redundant power connection ports 232, the power control system canadditionally include a power connection selection system. The powerconnection selection system (i.e., a power switch system) can functionto appropriately select at least one of the power connections for use.In the event of a failure or a drop in current and/or voltage, the powerconnection selection system transfers to use of a live power connection.The power connection selection system can be implemented through astatic circuit design that automatically changes based on configuredpower thresholds. The power connection selection system couldalternatively actively sense and control selection of a powerconnection. In this variation, use of a communication connection portcould be remotely monitored and/or controlled.

Related to this, the power control system can include a hotswapcontroller as discussed herein to manage the addition or removal of acamera module and/or a camera sub-system unit 202 while the system isrunning.

A distinct power port could additionally or alternatively be included ina camera module 200. In one variation, the camera module 200 can includea supplemental power port, which may be used in place of a powerconnection port 230 used in serial connection. In some cases, a branchor segment of a monitoring network 100 can surpass the power limits ofthe communication connection and so a supplemental power connection canbe used to supplement power supply. A power connection port 232 couldsimilarly be connected to a different power supply such that networkconnectivity may be continuous along a series, but multiple powersupplies are used supply power across sub-regions of the series ofcamera modules 200 as shown in FIG. 3.

For the communication connection port 234, the camera module 200preferably connects a wired network connection to the communicationcontrol system described above. The communication control system ispreferably configured to coordinate data communication of data collectedfrom the at least one camera of the camera module instance. Datacommunication can be either direction and can use any suitablecommunication protocol. The communication connection port 234 may be anethernet connection port with an RJ45 connector. Data communicated caninclude video media data, CV-derived metadata, operating parameters ofthe camera module, commands or data relayed to the camera module,software/firmware updates, and/or other suitable forms of data.

In one implementation, camera sub-system units 202 can additionallyinclude unit connection ports 236 so that they can be connected toconnection ports 230 exposed on the camera module 200 and connect toother camera sub-system units 202 as shown in FIGS. 14A and 14B.

The system may additionally include a wire management mechanism that canbe integrated with or used in combination with the camera modules. Thewire management mechanism functions to facilitate wire management whenconnecting multiple camera modules. In one variation, the wiremanagement mechanism can be a static structure that can be used to holdand store excess wiring. The wire management mechanism preferably holdsa small coil of wire near where the wire connects at the connection portas shown in FIG. 25. The wire management mechanism could additionallyfunction to reduce stress on a wire near the connector. In anotherimplementation, the system includes an extendable wire cartridge thathas a spring-loaded or manually wrapped coil of wire that can beconnected to the camera module 200. Preferably, a set of cartridges withdifferent lengths of wire may be provided such that someone connectingcan select a cartridge that is minimally sufficient to cover the wiredconnection. Excess wire can then be stored within the cartridge body.

Local and Remote Computing Systems

In addition to the camera modules 200 and connections to form themonitoring network, the system will generally include or interface withone or more power sources and computing systems. A power source can be apower supply that supplies the electrical current for powering thecamera modules. The computing system may include local computing systemsthat functions as a central on-premise computing resource. The localcomputing system can be a single master system but could alternativelybe multiple on-premise computing resources operating in cooperation. Alocal computing system could facilitate mass media storage and moreintensive computing capabilities. Operations that depend on significantcomputational power can be delegated to an on-premise computingresource. The local computing system can additionally manage the networkof camera modules. Additionally or alternatively, the system can includea remote computer resource, which functions as a cloud hosted computingresource. The remote computing resource can facilitate data sharingbetween different environments, enhancement of computer vision andmachine intelligence, remote monitoring of multiple sites and/or otherfeatures. An alternative implementation can have the camera modulesacting in fully distributed manner without dependence on a centralcomputing resource.

3. Method

The system preferably includes a number of operating modes thatfacilitate various features of the system. These features function toaddress various aspects of the system such as setting up the system in anew environment, operating the system in a new environment, andmaintaining the new system. In particular, various operating processesof the system can be applied to automatically calibrate the monitoringnetwork, calibrate a camera topology of the monitoring network 100within an environment, and manage media across the monitoring networkwithin the monitoring network.

Automatically Calibrating the Communication Network

A method for automatically calibrating the communication network of themonitoring network functions to make enrollment of a new camera module astreamlined process. The monitoring network is preferablyself-assembling and adaptive to changes. Adding a new camera module 200,removing a camera module 200, and/or reconfiguring a camera module canall trigger automatic reassignment of IP addresses. As discussed above,switches within the camera modules can use internal or external DHCP todistribute network configuration to the various components of themonitoring network.

Calibrating Camera Topology

A method for calibrating camera topology of the monitoring networkfunctions to enable the relative orientation and positioning of camerasto be at least approximately determined. Cameras can automaticallydetect proximity to other cameras, overlapping fields of view, and/orrelative positioning of observable fields of view. CV-based tracking ofobjects across multiple cameras can be used in determining relativeposition and orientation. Detection of a networking topology mayadditionally be used.

Calibrating camera topology can include calibrating a camera field ofview to an absolute position of the environment, which functions to maplocations in an environment to observable locations of a camera. In oneimplementation, a calibration beacon can be moved through theenvironment. The beacon preferably records its absolute position (e.g.,a GPS coordinate or a location coordinate relative to the inside of abuilding). The beacon additionally transmits or shows a visiblyobservable marker so that the absolute position can be associated withcameras that visually observe the beacon.

Calibrating the camera topology can be extended to include generating anenvironmental topology. In a variation, where the system includes objectclassification capabilities, the location of items, people, structures,and/or other features can be determined. As an exemplary application ofthe environmental topology. A store could generate a planogram of solditems and that planogram could be updated in substantially real-time.

Additionally, calibrating camera topology of the monitoring network caninclude reporting on health of camera topology. Reporting on the healthof the camera topology can include detecting holes in coverage (e.g.,locations where a camera does not have visibility). Reporting the healthof the camera topology can additionally include detecting unnecessarycamera redundancy at particular locations.

Managing Media within the Monitoring Network

A method of managing media within the monitoring network duringdistributed CV-based processing functions to augment data communicationand computational processes. As one objective, the method for mediamanagement preferably preserves observational data so that longitudinalanalysis of information can be performed. The preservation ofobservational data can allow for observations to be stored orrepresented in different forms.

As a treelike network with a trunk at a local computing system, themonitoring network preferably has connectivity to a local computingsystem. Distributed CV processing can be used to reduce the amount ofcommunicated media. Additionally, storing media locally can enable mediato be requested on demand from different camera modules. Managing mediapreferably includes dynamically tiering communication of a mediarepresentation. A media representation can be a media file of varyingmediums and formats but could additionally or alternatively includemetadata descriptions. The tiering of communications can be based oncamera prioritization. Camera prioritization can be influenced byprocessing of the image data (e.g., detecting a change or detecting aperson present), directions from nearby cameras (e.g., for trackingobjects between cameras), or directions from other resources (e.g.,local processing system prioritizes or deprioritizes a camera based onsome state change).

One implementation of dynamically tiering communication of a mediarepresentation can include: updating image data on scene change (e.g.,motion); communicating high quality media data when there is lowCV-processing confidence; communicating reduced quality media data withsupplemental metadata when there is moderate CV-processing confidence;communicating a metadata-only representation when there is highCV-processing confidence. Here a CV-processing confidence threshold isused as an exemplary condition, but alternative or additional conditionscould be used.

Similar tiering can also be applied to storage of observational data atdifferent locations of the monitoring network. For example, a cameramodule may increase its storage capacity by altering the format ofstored data to accommodate.

4. System and Method for Optical Enhancement

A system and method 300 for optical enhancement for ubiquitous videomonitoring functions to enable a network of cameras to be efficientlyinstalled and operated for collecting media, in particular, video andstill images.

The system and method 300 preferably applies a set of optical systems toalter optical imaging of distinct regions of an environment in acoordinated manner. Preferably, the system and method 300 can collectimage data for one or more target regions in an environment where thedifferent target regions are optically altered according to collectivegoals around working with the image. A first preferred target region isthe surface of a shelf, which may be holding various items like in astore. A second target region can be the walking space where customersmove and interact within the environment. Other target regions could becounters, tables, bins, conveyor belts of point of sale kiosks, uprightcomputer kiosks, and the like.

In some variations, the system and method 300 are used to monitor alarge structure or region with multiple cameras. For example, the faceof a shelf can be monitored with multiple cameras. The correctiveoptical systems 320 used in multiple cameras can enhance control overthe nature of monitoring this area. Corrective distortion and/or focalplane adjustment are two exemplary factors that can be adjusted for aparticular scenario. This can be of particular relevance when thecameras are mounted above a shelf.

Additionally or alternatively, a camera with a set of optical systemsmay be used to create distinct focus planes from a single camera unit.For example, a single camera mounted above a grocery store aisle asshown in FIG. 26, may have a first focus plane corrected for imaging ofa shelf and a second focus plane corrected for people and items movingacross the floor of the aisle.

The system is primarily described collecting video data. As used herein,‘video’ might mean a common video format such as NTSC, PLA, MPEG, 4K,etc. Video may also mean a sequence of individual still images whoseplace in time are monotonic. Those images may similarly be in a widerange of formats, including Raw, JPG, PDF, GIF, BMP, HEIF, HEVC, WebP,and/or other suitable media formats.

The system could additionally be applied to a single camera and mayadditionally alternatively be applied to other forms of image sensing.

As above, the system and method 300 may have particular applications insystems designed for ubiquitous monitoring across an environment.

The system and method 300 can additionally have general applicability tojust video or image surveillance or capture systems. For example, thesystem and method 300 could be employed to enable a camera system toefficiently take an image of a shelf of items periodically for anysuitable type of application.

The system and method 300 has particular applications to imaging a shelfor vertically arranged items, as is a common situation in stores. Morespecifically, the system and method 300 can be used for enhancingimaging while capturing the shelf, items on the shelf, or other targetregion from an off-angle perspective as is common when cameras aremounted with an aerial perspective above the target region. Otherscenarios or target subjects may alternatively be used, but herein theshelf use-case is used as the primary example.

As one potential benefit, the system and method 300 can produce a camerasystem that has improved focus and/or distortion effects. In manycommerce environments, camera systems are positioned above the region ofinterest or in other out-of-the-way regions of the environment. As such,the regions of interest for video monitoring often happen along planesnon-parallel to the sensor planes of the camera system. Imaging ofshelves from a ceiling mounted camera system is one such example. Focusof items stacked in the shelves may normally have different levels offocus. The system and method 300 may correct the focus plane to moreclosely align with the plane of a shelf or other target region. Keystonedistortion of a shelf face or other target region additionally willgenerally occur with traditional approaches. The system and method 300may reduce perspective distortion for a target region like a shelf face.This can make the pixel density across the shelf face to be uniform orat least more uniform.

As a related potential benefit, perspective correction may be appliedacross multiple cameras with configured arrangement such that theimaging of the target region is controlled in a more uniform manner. Asopposed to an arrangement of cameras capturing shelf images withkeystoned overlaps of shelf images (e.g., manifesting as trapezoidaland/or triangular overlap regions), the system and method 300 can reduceor correct distortion such that arrangement of cameras can be set basedon more parallel imaging of the target region as shown in FIGS. 18B,18C, and 18D. As a result, camera coverage resulting from overlap is amore uniform (e.g., rectangular regions of overlap). Camera coverage canthereby be modified based on desired types of coverage like 100%coverage, 200% coverage, 300% coverage, or other suitable forms ofcoverage.

As another potential benefit, the system and method 300 can enableindividual cameras to have customized imaging of distinct target regionswherein the focus plane and/or distortion correction is applied in atleast one region of the image plane. This can be used in situationswhere a shelf could be monitored with perspective correction and focusplane alignment while the ground region can be monitored with a focaldistance customized for tracking people.

In the variations described above and below, the system and method 300may have the added benefit of more efficient use of cameras whereoptical systems are customized for monitoring specific scenarios withina store. When deploying a ubiquitous imaging system across anenvironment, especially a large environment like a store, the number ofcameras and devices to appropriately observe the space may be a highnumber. Camera efficiency can be critical in reducing physical componentcost as well as reducing the amount of data for processing and storage.As such, more efficient camera coverage, better camera coveragealignment, more uniform pixel density of objects of target regions, andother factors resulting in part from the optical system of the systemand method 300 may all function to reduce camera numbers.

As one example, a system employing the optical enhancement may reducecamera count by 25%-50% which saves on physical compute resources todrive the cameras which includes smaller footprint on processing, datastorage, enclosure size/complexity, data and power connectivity.

A system for optical enhancement for ubiquitous video monitoring of apreferred embodiment preferably includes a set of optical systemsconfigured to customize optical imaging for coordinated monitoring ofdifferent regions. In one variation, the different regions may besub-regions of one continuous subject of interest. In another variation,different regions like a shelf and the ground may have differentexpected monitoring.

The system may primarily function to coordinate the collection of imagedata from different regions in a manner that is customized to thechallenges for CV-driven applications. In particular, the system hasapplications to imaging within a place of commerce like a grocery store,convenience shop, or other suitable type of store where there areshelves of items lined along a defined plane not parallel to the imagesensor of a camera. The system preferably includes a multi-cameravariation and a split-view variation, which may be used individually orin combination.

Multi-Camera Variation

In a first preferred variation, the system preferably includes: a set ofcameras 310, the cameras being configured with a relative cameraarrangement; and each camera including at least one corrective opticalsystem 320 as shown in FIG. 17. The multi-camera variation, functions tocoordinate the optical modification of imaging by the cameras to accountfor the relationship of the images captured across the cameras 310. Forexample, the multi-camera variation can have particular applicationswhen used in combination with a monitoring network and/or camera moduledescribed above. Capturing images or video of a subject-of-interestacross an environment may be achieved with higher level of camera usageefficiency. For example, a wider area of the shelf could be monitoredwith a fewer number of cameras.

The variation can function to perform high resolution video monitoringof a structure from an off-axis camera system using multiple cameras.This variation has particular benefit for capturing images of longcontinuous structures with surfaces that are non-parallel to the camerasuch as a shelf when monitored from a camera mounted above the height ofthe shelf. Through this system variation, a long shelf face can bemonitored with substantially uniform image resolution across the targetregion (e.g., shelf face). For example, the pixel density of objects atthe bottom of the shelf, after perspective correction, can besubstantially similar pixel density of objects at the top shelf. Morespecifically, this variation can perform video monitoring of a targetregion like a shelf face with substantially uniform image resolutionacross the target region.

The multi-camera variation may additionally have wider applications inother camera array applications. Herein, the multi-camera variationprimarily addresses the application to imaging within a commercial spaceand in particular monitoring a shelf from a camera position that isdisplaced horizontally in front of the shelf face and above the shelfface (or region of interest on the shelf face). Other suitable scenarioscould alternatively use this multi-camera variation.

A camera 310 functions to capture an image of the environment. Thecamera 310 preferably includes at least a camera sensor. In onevariation, the cameras 310 may be integrated into a single connectedstructure. As one implementation, the set of cameras 310 may beintegrated into a camera module as discussed above sometimes grouped ascamera pairs. Furthermore, multiple camera modules and/or othermulti-camera devices could be aligned, connected, or otherwise arrangedso as the set of cameras 310 includes multiple distinct devices thatoperate in combination. Accordingly, in another implementation, thesystem may include a monitoring network of interconnected camera modulesas discussed above where each camera module comprises of distinct subsetof cameras from the set of cameras,

As one preferred arrangement, the set of cameras are arranged with acorresponding angle orientation (relative to expected camera mountingconfiguration). The set of cameras may be directed with defined sensorplanes in a coplanar arrangement. More specifically, the defined sensorplanes of the cameras can be coplanar within some expected tolerances.For example, a line of cameras may be mounted along a line and directedin the same general orientation towards different positions of a shelf.In another variation, the arrangement of the cameras may be such thatthe cameras with tangential angle orientations along some path like anarc or an arbitrary path. For example, a ring of cameras maycircumscribe a circular shelf.

Video or image data is collected from the set of cameras 310 and ispreferably communicated to a centralized processing system.

The corrective optical system 320 functions to alter or transform theimaging of the environment. The transformation is preferably customizedto the objectives in the environment. This transformation may involveperspective correction, focal plane adjustment, setting of focaldistance and/or other suitable field of view adjustments.

In the case of monitoring shelves, the corrective optical system 320 ispreferably configured with perspective correction. A shelf when capturedby a camera above the shelf face will be captured with a perspectivedistortion that results in the top of the shelf having more pixeldensity resolution (e.g., appearing larger) and the bottom of the shelfhaving lower pixel density resolution (e.g., appearing smaller).

The corrective optical system 320 can be a perspective correctionoptical system. In one implementation, the perspective correctionoptical system can be a shift lens optical system or more preferably atilt-shift lens optical system. Shifting and/or tilting of a lens canproduce corrective distortion that can counteract normal distortions.This may be a single lens optical system but could alternatively includemultiple stage optical system. A captured image of a shelf can becorrected with reduce keystoning as shown in FIG. 8D or more preferablysubstantially removed keystoning (e.g., vertical edges of shelf areparallel within 5 degrees) as shown in FIG. 18B. The perspectivecorrection could be adjusted for various situations. For example, ifmonitoring a long bin of items, the perspective correction opticalsystem could correct for keystoning of the horizontal surface of thebin. If monitoring an area that is at a non-perpendicular angle (e.g.,in a gravity-defined orientation) such as seats in an auditorium,perspective correction could similarly be applied for diagonally alignedsurfaces.

As one set of enhancements offered through perspective correction, theperspective transformation can be configured to reduce perspectivedistortion of a target subject like a shelf face. In the case of a shelfface, the defined vertical sides of a resulting captured image of theshelf face will have angles less than two degrees from vertical. Thoughany suitable level of correction may alternatively be used.

Similarly, the imaging resolution of a shelf (or target region) arealtered as a result of perspective correction, which can improve imageresolution uniformity across a target subject. For example, the capturedimage of FIG. 18B will have objects at the bottom shelf imaged withsimilar pixel densities compared to objects on the top shelf.

One potential benefit of the use of a corrective optical system 320 andmore specifically at least a perspective correction optical system canbe that camera coverage of a shelf face or other suitable target regionmay be better managed. When the perspective correction optical system isapplied across a set of arranged cameras, the image planes can betransformed to have adjacent overlap that is controlled throughcoordination of optical correction system and camera arrangement. Morespecifically, a tilt-shift optical system can be configured to apply atleast a perspective transformation, while the relative cameraarrangement and the perspective transformation are cooperativelyconfigured to image of a target subject (e.g., a shelf face).

The adjacent overlap (e.g., overlap of camera coverage by neighboringcameras) can be controlled for consistent camera coverage across thevertical face of the shelf face. When not corrected, keystoning canresult in different levels of camera coverage. As shown in FIG. 18A, thecamera coverage of a shelf, when viewed by a camera module 200 that ismounted above and in front of a shelf face, can have regions of coveragewith small redundancy and very high redundancy when not corrected. Inthis figure regions of darker color indicate the degree of cameracoverage overlap or redundancy. In this example, bottom regionssometimes have 400%, 300%, 200%, or 100% camera coverage while the topof the shelf sees 200% or 100%. While such redundancy may be permissiblein many situations, some scenarios may benefit from more control ofcamera coverage.

With correction, the image plane overlap at the target subject ispreferably rectangular or is at least more uniform (e.g., reducedkeystoning) compared to image plane without perspective correction.

The optical system and the relative camera arrangement can becooperatively configured for different levels of adjacent-overlap ofcamera coverage. More specifically, the perspective transformation ofthe optical system and the relative camera arrangement is configuredfrom some level of adjacent camera overlap. One targeted overlap profilemay target 100% camera coverage, 200% camera coverage, 300% cameracoverage, and/or any suitable variation. The configured overlap may bebased on expected camera mounting position. In the shelf use case, theset of cameras are generally mounted above and in front of shelf face.

In a minimizing variation, coverage of a target subject may be designedto have 100% coverage of the target subject. Ideal adjacent overlap canbe configured for perfect alignment with zero overlap. However, innormal conditions, some amount of adjacent overlap may be preferable toallow for variability. Accordingly, the adjacent overlap between isgenerally configured to be less than 25% and in some implementations maybe configured to be 10% or less. As shown in FIG. 18B, the spacedarrangement of cameras can be configured to align the adjacent cameraoverlap to be less than a quarter of the imaged shelf face. This canfunction to minimize redundancy, which can reduce the number of cameras.

As shown in FIG. 18C, the spaced arrangement of cameras mayalternatively be configured to align the adjacent camera overlap topromote a particular degree of camera coverage redundancy. To achievecamera redundancy, the relative camera arrangement and the perspectivetransformation can be cooperatively configured for at least 50% adjacentcamera overlap in imaging a target subject like a shelf face. Theadjacent camera overlap may be limited to no more than a certainthreshold like 60%. In this example, each region of covered shelf isviewable by two cameras (e.g., 200% coverage). Even without perfectdistortion correction as shown in FIG. 18D, a corrective shift opticalsystem can enhance the control of camera coverage.

The corrective optical system 320 may additionally or alternatively beconfigured with focal plane adjustment wherein the focal plane can bereoriented to align with a desired target subject. In a standard imagingapproach, objects on a shelf will have different distances from thecamera and thereby will generally not all be absolutely focused. Thefocal plane adjustment is preferably achieved through a tilted lens andmore preferably a tilt-shift lens. This may be a single lens opticalsystem but could alternatively include a multiple stage optical system.Focal plane adjustment is preferably used in combination withperspective correction discussed above. Preferably, the focal plane isadjusted to more closely align with a defined surface of the shelf face.In one implementation, a tilt-shift optical system is configured toalign the focal plane within ten degrees of the shelf face, though anysuitable degree of alignment may be performed. As a result, the itemsvisible at the front of the shelf may be having more uniform levels offocus.

The focal plane adjustment correction could be adjusted for varioussituations. In some variations, the target subject may be defined as ahorizontal surface region, where the set of cameras are mounted aboveand off-center from a horizontal surface region. Correspondingadjustments may be made to address such imaging for each camera andoptical system and across camera arrangement. For example, if monitoringa long bin of items, the focal plane can be adjusted to align withhorizontal surface of the bin. In another variation, the target subjectmay be defined as a diagonal surface region. In other words, the targetsubject can be along a plane non parallel or perpendicular angle (e.g.,in a gravity-defined orientation) such as seats in an auditorium. In oneexample, the focal plane could similarly be aligned with the definedangular plane of the target region.

Split-View Variation

In a second preferred variation, the system preferably includes a camera300 with a split-view optical system 330 as shown in FIG. 27. Asplit-view optical system 330 as described herein has at least tworegions 331 and 332 of an image captured with distinct opticaltransformations.

This split-view variation can function to customize focus distance,focal plane orientation, perspective distortion correction, or otheroptical properties within subregions of an image. A split-view variationis preferably useful in efficiently utilizing area of an image sensorfor capturing different regions of interest in the near proximity.

The split-view variation may be used individually, in combination withthe multi-camera variation, and/or more specifically in combination witha monitoring network of camera modules as discussed above. Thesplit-view optical system 330 is preferably a variation of a correctiveoptical system 300. Accordingly, the corrective optical system 320 withthe multi-camera variation may further use a split-view optical system330.

Within a split-view optical system 330, different “views” or regions ofthe field of view are transformed in different ways. The types oftransformations can include perspective correction, focal planeadjustment, field of view adjustment, redirection, focal distanceadjustment, and/or other suitable optical transformations. These opticaltransformations can be combined in any suitable combination.

The split-view optical system 330 preferably produces a set of imagedregions. In one preferred variation, a camera 300 can be configured tocapture at least a first imaged region and a second imaged region fromthe split-view optical system 330. In other variations, a third imagedregion or any suitable number of imaged regions can be produced by thesplit-view optical system 330. An imaged region as used here refers tothe sub-region of image data resulting from optical transformationapplied to a portion of the field of view of a camera sensor.

The imaged regions resulting from distinct optical transformations mayhave distinct transitions or edges. The resulting image may be processedwith each imaged region treated as a distinct image where the relevantimage portion can be isolated from the source image. Alternatively, anoptical transformation may be graduated so as to provide gradualtransitions. The regions may be image data from distinct areas in theenvironment 331 and 332 as shown in FIG. 28. Alternatively, a region mayinclude image data redundant from another region. For example, the floorregion may include image that also includes shelf image data as shown inFIG. 29.

An optical sub-system of a split-view optical system 330 for perspectivecorrection and/or focal plane adjustment can preferably apply the abovevariations. A tilt-shift optical sub-system 340 is preferably used toapply at least one of these corrections. The split-view optical system330 preferably includes at least one tilt-shift optical sub-system 340.When used in imaging a shelving aisle, the tilt-shift optical sub-system340 can be localized to an image region isolated to primarily a shelfface.

An optical sub-system can be used to adjust the field of view. In onevariation, a region may have narrowed field of view to focus on a regionof interest. In another variation, a region may have widening field ofview. A wide field of view optical subsystem may be useful insituations, where an area in the environment should be monitored is notdirectly next to the other imaged areas, and can be viewed with reducedimage resolution. For example, a secondary shelf face.

An optical sub-system may apply a redirection transformation wherein amirror or other optical system redirects the optical path for the regionto another space.

In the shelf aisle use case, the camera and split-view optical system330 may be configured to capture a first shelf face, a floor region withuser traffic, and/or possibly a second shelf face. Each region may haveparticular optical transformations.

Preferably, at least one of the shelves is monitored with an objectiveof item identification and item interaction classification anddetection. For example, the image region capturing the first shelf ispreferably suitable for detecting the items on the shelf and/or theevent of a customer removing or adding an item to the shelf. In onepreferred variation, an imaged region capturing a shelf face isoptically transformed by a tilt-shift optical sub-system 340 of thesplit-view optical system 330.

As shown in FIGS. 26 and 28, a tilt-shift optical sub-system 340optically transforms a portion of the cameras field of view. This maygenerally be used for capturing shelf face images. There may be avariety of options in using the other imaged regions.

In one variation, the at least second imaged region can be opticallytransformed in a way giving it an effective focal plane different fromthe first imaged region. The focal plane could be further transformedthrough a second tilt-shift optical sub-system 340.

In another variation, the split-view optical system 330 can include amirror that is configured to form a second imaged region by redirectinga sub-region of the field of view as shown in FIGS. 26 and 28. This maybe used to redirect the field of view to a more useful region such as asecond shelf face, the floor surface, or any suitable region. The focaldistance and/or focal plane of the second region could also be adjustedthrough further optical transformations. Similarly, the field of view orother optical properties could similarly be set for the second imagedregion with an optical sub-system of the split-view optical system 330.

In one variation, the split-view optical system 330 may be used so thata camera can monitor two opposing shelves, which is generally applicablewhen the camera is mounted between and above two opposing shelf faces.In this variation, a camera can be configured to capture a first imagedregion and a second imaged region from the split-view optical system330. The first imaged region preferably captures a first shelf face, andthe second imaged region preferably captures a second shelf face. Thefirst imaged region from the split-view optical system 330 can beoptically transformed through a tilt-shift optical sub-system 340configured to apply perspective correction of the first shelf face.

The second shelf, which may be opposing the first shelf, may similarlybe monitored with corresponding objectives. In one variation, at leastone shelf face may be monitored with an objective of detecting generalshelf interaction. The location or region of interaction mayadditionally be detected. For example, a second shelf may be monitoredwith the intended objective of detecting when a customer interacts withthat shelf, where further classification of this event may be left toanother camera with a different perspective configured for itemclassification and event classification.

In another variation, the split-view optical system 330 may be used sothat a camera can monitor a shelf and a floor region. This is generallyapplicable when the camera is mounted in front of and above a shelfface. In this variation, each camera can be configured to capture afirst imaged region and a second imaged region from the split-viewoptical system 330. The first imaged region preferably captures theshelf face, and the second imaged region preferably captures the floorsurface. The first imaged region from the split-view optical system 330can be optically transformed through a tilt-shift optical sub-system 340configured to apply perspective correction of the shelf face. The secondregion may be altered in various ways to appropriately sense activity onor near the floor.

Preferably, the second imaged region is preferably optically transformedby an optical subsystem that is configured to adjust the focal planesubstantially parallel to a floor surface. This may function to monitorwhere people move through an environment. A floor imaged region 333 maybe further customized so that the focal plane is at an approximateheight of people's faces as shown in FIG. 29. For example, the focalplane of this imaged region 333 may be set to be at least five feet.

The different regions may be arranged so that they are imaged as asequence of rectangular images collected on one source image. Thedifferent regions could alternatively be optically transformed so thatregions are reflected in different quadrants of a source image or in anysuitable location.

The systems and methods of the embodiments can be embodied and/orimplemented at least in part as a machine configured to receivecomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated with apparatusesand networks of the type described above. The computer-readable mediumcan be stored on any suitable computer readable media such as RAMs,ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives,floppy drives, or any suitable device. The computer-executable componentcan be a processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

We claim:
 1. An imaging system comprising of: a set of cameras, thecameras having a configured relative camera arrangement; and each cameraof the set of cameras comprising at least one corrective optical system.2. The system of claim 1, wherein the corrective optical systemcomprises of at least one tilt-shift optical system.
 3. The system ofclaim 2, wherein the tilt-shift optical system is configured to apply atleast perspective transformation; and wherein the relative cameraarrangement and the perspective transformation are cooperativelyconfigured to imaging of a target object.
 4. The system of claim 3,wherein the target object is a shelf face; and wherein the relativecamera arrangement and the perspective transformation are cooperativelyconfigured for less than 10% adjacent camera overlap in imaging of theshelf face when the set of cameras are mounted above and in front of theshelf face.
 5. The system of claim 3, wherein the target object is shelfface; and wherein the relative camera arrangement and the perspectivetransformation are cooperatively configured for less than at least 50%adjacent camera overlap in imaging of the shelf face when the set ofcameras are mounted above and in front of the shelf face.
 6. The systemclaim 3, wherein the set of cameras are mounted above and in front of ashelf face; and wherein the perspective transformation is configured toreduce perspective distortion of the shelf face wherein defined verticalsides of a captured image of the shelf face has an angle less than twodegrees from vertical.
 7. The system claim 3, wherein the tilt-shiftoptical system is further configured to align the focal plane within tendegrees of the shelf face.
 8. The system of claim 3, wherein the targetobject is defined horizontal surface region; wherein the set of camerasare mounted above and off-center from horizontal surface region.
 9. Thesystem of claim 3, wherein the target object is a defined diagonalsurface region.
 10. The system of claim 3, wherein the set of camerasare integrated into a camera module housing multiple cameras.
 11. Thesystem of claim 3, further comprising a monitoring network ofinterconnected camera modules; wherein each camera module comprises ofdistinct subset of cameras from the set of cameras.
 12. The system ofclaim 1, wherein at least a subset of corrective optical systemscomprises a split-view optical system.
 13. The system of claim 12,wherein each camera is configured to capture at least a first imagedregion and a second imaged region from the split-view optical system;wherein the first imaged region from the split-view optical system isoptically transformed through a tilt-shift optical sub-system.
 14. Thesystem of claim 13, wherein the second imaged region has an effectivefocal plane different from the first imaged region.
 15. The system ofclaim 13, wherein the split-view optical system comprises a mirrorconfigured to form the second imaged region by redirecting a sub-regionof the field of view.
 16. The system of claim 12, wherein each cameraadditionally captures a third imaged region from the split-view opticalsystem.
 17. The system of claim 12, wherein the cameras are mounted infront of and above a shelf face; and wherein each camera is configuredto capture a first imaged region and a second imaged region from thesplit-view optical system, the first imaged region containing the shelfface, and the second imaged region containing the floor surface; andwherein the first imaged region from the split-view optical system isoptically transformed through a tilt-shift optical sub-system configuredto apply perspective correction of the shelf face.
 18. The system ofclaim 17, wherein the second imaged region from the split-view opticalsystem is optically transformed by an optical subsystem configured toadjust the focal plane substantially parallel to a floor surface. 19.The system of claim 18, wherein the optical sub-system orients the focalplane of the second imaged region to at least five feet above the floorsurface.
 20. The system of claim 12, wherein the cameras are mountedbetween and above two opposing shelf faces; and wherein each camera isconfigured to capture a first imaged region and a second imaged regionfrom the split-view optical system, the first imaged region containing afirst shelf face, and the second imaged region containing a second shelfface; and wherein the first imaged region from the split-view opticalsystem is optically transformed through a tilt-shift optical sub-systemconfigured to apply perspective correction of the first shelf face.